Aerospike Cluster Health Assessment

This report provides a technical analysis of the common-edge_va6prod cluster, derived from the telemetry bundle aws-common.collect_info_20260120_230014.tgz.

Vital Statistic	Value
Node Count	5 Nodes
Analysis Date	2026-02-08 21:48
Delete Not Found	50,720,801 Events

Executive Summary

Analysis of this snapshot identifies the cluster as consisting of 5 nodes. The primary observation from this period is the detection of 50,720,801 “Delete Not Found” events.

TAM Assessment Overview

This diagnostic focuses on configuration symmetry, performance outliers, and underlying infrastructure support. Review the “Observations and Remediation” section below for specific node-level actions.

Cluster Status Overview

Config Symmetry

Detected 365 configuration drifts across the cluster.

Delete Not Found

Detected 50,720,801 ‘Delete Not Found’ events across 5 nodes.

Config Drift

The static aerospike.conf file was not found in the collectinfo bundle.

Performance and Utilization

Disk Usage % by Node

Client Connections

Observations and Remediation

1.b: ENA Support Check

ENA Telemetry Missing: OS-level statistics were not found in this bundle.

Recommendation: To retrieve this data, run ‘collectinfo’ as root or use ‘asadm –lsmod –ethtool’. Manually verify with: ‘ethtool -i eth0’.

1.c: Version Consistency

Telemetry not yet ingested.

Recommendation: Review cluster telemetry and application logs for further investigation.

3.a: Config Symmetry

Detected 365 configuration drifts across the cluster.

Recommendation: Check the Technical Details section to identify which specific parameters (e.g. memory, timeouts) differ.

3.b: Config Drift

The static aerospike.conf file was not found in the collectinfo bundle.

Recommendation: Cannot perform drift analysis (Runtime vs. Static). To enable this check in the future, run: ‘asadm -e “collectinfo –cf /etc/aerospike/aerospike.conf”’

4.b: Read Not Found

Detected 4,068 ‘Read Not Found’ events. This is often normal for application cache-miss workflows.

Recommendation: If the application expects all records to exist, investigate potential expiration or eviction issues.

4.c: Delete Not Found

Detected 50,720,801 ‘Delete Not Found’ events across 5 nodes.

Recommendation: This suggests the application is attempting to delete records that do not exist (Blind Deletes). Review application logic.

Appendix: All Tests Performed

Check Name	Status
1.a: Service Error Skew	✅ PASS
1.b: ENA Support Check	⚠️ INFO
1.c: Version Consistency	⚠️ INFO
2.a: SIndex on Flash	✅ PASS
2.b: Sprig Limit Warning	✅ PASS
2.c: Storage Deadlock Risk	✅ PASS
2.d: Disk HWM Check	✅ PASS
2.e: Memory HWM Check	✅ PASS
3.a: Config Symmetry	⚠️ WARNING
3.b: Config Drift	⚠️ ⚠️ DATA MISSING
4.a: Hot Key Detection	✅ PASS
4.b: Read Not Found	⚠️ INFO
4.c: Delete Not Found	⚠️ WARNING

TAM Technical Details

Node Statistics (Sample)

node_id	metric	value
10.94.173.82:3000	service.info_timeout	0.0
10.94.173.43:3000	service.cluster_max_compatibility_id	15.0
10.94.173.12:3000	service.heartbeat_connections_closed	12.0
10.94.173.12:3000	service.batch_index_created_buffers	127.0
10.94.173.73:3000	service.client_connections_opened	55265206.0
10.94.173.82:3000	service.fabric_ctrl_recv_rate	0.0
10.94.173.12:3000	service.info_timeout	0.0
10.94.173.43:3000	service.heap_mapped_kbytes	2611200.0
10.94.173.40:3000	service.cluster_clock_skew_ms	0.0
10.94.173.82:3000	service.early_tsvc_from_proxy_error	0.0

Aerospike Health Analyzer | Version: {python} PROJECT_VERSION | Generated: {python} GEN_DATE